Amazon Fine Food Reviews Analysis¶
Data Source: https://www.kaggle.com/snap/amazon-fine-food-reviews
EDA: https://nycdatascience.com/blog/student-works/amazon-fine-foods-visualization/
The Amazon Fine Food Reviews dataset consists of reviews of fine foods from Amazon.
Number of reviews: 568,454
Number of users: 256,059
Number of products: 74,258
Timespan: Oct 1999 - Oct 2012
Number of Attributes/Columns in data: 10
Attribute Information:
- Id
- ProductId - unique identifier for the product
- UserId - unqiue identifier for the user
- ProfileName
- HelpfulnessNumerator - number of users who found the review helpful
- HelpfulnessDenominator - number of users who indicated whether they found the review helpful or not
- Score - rating between 1 and 5
- Time - timestamp for the review
- Summary - brief summary of the review
- Text - text of the review
Objective:¶
Given a review, determine whether the review is positive (rating of 4 or 5) or negative (rating of 1 or 2).
[Q] How to determine if a review is positive or negative?
[Ans] We could use Score/Rating. A rating of 4 or 5 can be cosnidered as a positive review. A rating of 1 or 2 can be considered as negative one. A review of rating 3 is considered nuetral and such reviews are ignored from our analysis. This is an approximate and proxy way of determining the polarity (positivity/negativity) of a review.
[1]. Reading Data¶
[1.1] Loading the data¶
The dataset is available in two forms
- .csv file
- SQLite Database
In order to load the data, We have used the SQLITE dataset as it is easier to query the data and visualise the data efficiently.
Here as we only want to get the global sentiment of the recommendations (positive or negative), we will purposefully ignore all Scores equal to 3. If the score is above 3, then the recommendation wil be set to "positive". Otherwise, it will be set to "negative".
In [1]:
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")
import sqlite3
import pandas as pd
import numpy as np
import nltk
import string
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import confusion_matrix
from sklearn import metrics
from sklearn.metrics import roc_curve, auc
from nltk.stem.porter import PorterStemmer
import re
# Tutorial about Python regular expressions: https://pymotw.com/2/re/
import string
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from gensim.models import Word2Vec
from gensim.models import KeyedVectors
import pickle
from tqdm import tqdm
import os
In [2]:
# using SQLite Table to read data.
con = sqlite3.connect('database.sqlite')
# filtering only positive and negative reviews i.e.
# not taking into consideration those reviews with Score=3
# SELECT * FROM Reviews WHERE Score != 3 LIMIT 500000, will give top 500000 data points
# you can change the number to any other number based on your computing power
# filtered_data = pd.read_sql_query(""" SELECT * FROM Reviews WHERE Score != 3 LIMIT 500000""", con)
# for tsne assignment you can take 5k data points
filtered_data = pd.read_sql_query(""" SELECT * FROM Reviews WHERE Score != 3 LIMIT 50000""", con)
# Give reviews with Score>3 a positive rating(1), and reviews with a score<3 a negative rating(0).
def partition(x):
if x < 3:
return 0
return 1
#changing reviews with score less than 3 to be positive and vice-versa
actualScore = filtered_data['Score']
positiveNegative = actualScore.map(partition)
filtered_data['Score'] = positiveNegative
print("Number of data points in our data", filtered_data.shape)
filtered_data.head(3)
Out[2]:
In [3]:
display = pd.read_sql_query("""
SELECT UserId, ProductId, ProfileName, Time, Score, Text, COUNT(*)
FROM Reviews
GROUP BY UserId
HAVING COUNT(*)>1
""", con)
In [4]:
print(display.shape)
display.head()
Out[4]:
In [5]:
display[display['UserId']=='AZY10LLTJ71NX']
Out[5]:
In [6]:
display['COUNT(*)'].sum()
Out[6]:
[2] Exploratory Data Analysis¶
[2.1] Data Cleaning: Deduplication¶
It is observed (as shown in the table below) that the reviews data had many duplicate entries. Hence it was necessary to remove duplicates in order to get unbiased results for the analysis of the data. Following is an example:
In [7]:
display= pd.read_sql_query("""
SELECT *
FROM Reviews
WHERE Score != 3 AND UserId="AR5J8UI46CURR"
ORDER BY ProductID
""", con)
display.head()
Out[7]:
As it can be seen above that same user has multiple reviews with same values for HelpfulnessNumerator, HelpfulnessDenominator, Score, Time, Summary and Text and on doing analysis it was found that
ProductId=B000HDOPZG was Loacker Quadratini Vanilla Wafer Cookies, 8.82-Ounce Packages (Pack of 8)
ProductId=B000HDL1RQ was Loacker Quadratini Lemon Wafer Cookies, 8.82-Ounce Packages (Pack of 8) and so on
It was inferred after analysis that reviews with same parameters other than ProductId belonged to the same product just having different flavour or quantity. Hence in order to reduce redundancy it was decided to eliminate the rows having same parameters.
The method used for the same was that we first sort the data according to ProductId and then just keep the first similar product review and delelte the others. for eg. in the above just the review for ProductId=B000HDL1RQ remains. This method ensures that there is only one representative for each product and deduplication without sorting would lead to possibility of different representatives still existing for the same product.
In [8]:
#Sorting data according to ProductId in ascending order
sorted_data=filtered_data.sort_values('ProductId', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
In [9]:
#Sorting data according to Time in ascending order
sorted_data=sorted_data.sort_values('Time', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
In [10]:
#Deduplication of entries
final=sorted_data.drop_duplicates(subset={"UserId","ProfileName","Time","Text"}, keep='first', inplace=False)
final.shape
Out[10]:
In [11]:
#Checking to see how much % of data still remains
(final['Id'].size*1.0)/(filtered_data['Id'].size*1.0)*100
Out[11]:
Observation:- It was also seen that in two rows given below the value of HelpfulnessNumerator is greater than HelpfulnessDenominator which is not practically possible hence these two rows too are removed from calcualtions
In [12]:
display= pd.read_sql_query("""
SELECT *
FROM Reviews
WHERE Score != 3 AND Id=44737 OR Id=64422
ORDER BY ProductID
""", con)
display.head()
Out[12]:
In [13]:
final=final[final.HelpfulnessNumerator<=final.HelpfulnessDenominator]
In [14]:
#Before starting the next phase of preprocessing lets see the number of entries left
print(final.shape)
#How many positive and negative reviews are present in our dataset?
final['Score'].value_counts()
Out[14]:
[3] Preprocessing¶
[3.1]. Preprocessing Review Text¶
Now that we have finished deduplication our data requires some preprocessing before we go on further with analysis and making the prediction model.
Hence in the Preprocessing phase we do the following in the order below:-
- Begin by removing the html tags
- Remove any punctuations or limited set of special characters like , or . or # etc.
- Check if the word is made up of english letters and is not alpha-numeric
- Check to see if the length of the word is greater than 2 (as it was researched that there is no adjective in 2-letters)
- Convert the word to lowercase
- Remove Stopwords
- Finally Snowball Stemming the word (it was obsereved to be better than Porter Stemming)
After which we collect the words used to describe positive and negative reviews
In [15]:
# printing some random reviews
sent_0 = final['Text'].values[0]
print(sent_0)
print("="*50)
sent_1000 = final['Text'].values[1000]
print(sent_1000)
print("="*50)
sent_1500 = final['Text'].values[1500]
print(sent_1500)
print("="*50)
sent_4900 = final['Text'].values[4900]
print(sent_4900)
print("="*50)
In [16]:
# remove urls from text python: https://stackoverflow.com/a/40823105/4084039
sent_0 = re.sub(r"http\S+", "", sent_0)
sent_1000 = re.sub(r"http\S+", "", sent_1000)
sent_150 = re.sub(r"http\S+", "", sent_1500)
sent_4900 = re.sub(r"http\S+", "", sent_4900)
print(sent_0)
In [17]:
# https://stackoverflow.com/questions/16206380/python-beautifulsoup-how-to-remove-all-tags-from-an-element
from bs4 import BeautifulSoup
soup = BeautifulSoup(sent_0, 'lxml')
text = soup.get_text()
print(text)
print("="*50)
soup = BeautifulSoup(sent_1000, 'lxml')
text = soup.get_text()
print(text)
print("="*50)
soup = BeautifulSoup(sent_1500, 'lxml')
text = soup.get_text()
print(text)
print("="*50)
soup = BeautifulSoup(sent_4900, 'lxml')
text = soup.get_text()
print(text)
In [18]:
# https://stackoverflow.com/a/47091490/4084039
import re
def decontracted(phrase):
# specific
phrase = re.sub(r"won't", "will not", phrase)
phrase = re.sub(r"can\'t", "can not", phrase)
# general
phrase = re.sub(r"n\'t", " not", phrase)
phrase = re.sub(r"\'re", " are", phrase)
phrase = re.sub(r"\'s", " is", phrase)
phrase = re.sub(r"\'d", " would", phrase)
phrase = re.sub(r"\'ll", " will", phrase)
phrase = re.sub(r"\'t", " not", phrase)
phrase = re.sub(r"\'ve", " have", phrase)
phrase = re.sub(r"\'m", " am", phrase)
return phrase
In [19]:
sent_1500 = decontracted(sent_1500)
print(sent_1500)
print("="*50)
In [20]:
#remove words with numbers python: https://stackoverflow.com/a/18082370/4084039
sent_0 = re.sub("\S*\d\S*", "", sent_0).strip()
print(sent_0)
In [21]:
#remove spacial character: https://stackoverflow.com/a/5843547/4084039
sent_1500 = re.sub('[^A-Za-z0-9]+', ' ', sent_1500)
print(sent_1500)
In [22]:
# https://gist.github.com/sebleier/554280
# we are removing the words from the stop words list: 'no', 'nor', 'not'
# <br /><br /> ==> after the above steps, we are getting "br br"
# we are including them into stop words list
# instead of <br /> if we have <br/> these tags would have revmoved in the 1st step
stopwords= set(['br', 'the', 'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've",\
"you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', \
'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their',\
'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', \
'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', \
'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', \
'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after',\
'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further',\
'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\
'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very', \
's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', \
've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn',\
"hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn',\
"mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", \
'won', "won't", 'wouldn', "wouldn't"])
In [23]:
# Combining all the above stundents
from tqdm import tqdm
preprocessed_reviews = []
# tqdm is for printing the status bar
for sentance in tqdm(final['Text'].values):
sentance = re.sub(r"http\S+", "", sentance)
sentance = BeautifulSoup(sentance, 'lxml').get_text()
sentance = decontracted(sentance)
sentance = re.sub("\S*\d\S*", "", sentance).strip()
sentance = re.sub('[^A-Za-z]+', ' ', sentance)
# https://gist.github.com/sebleier/554280
sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
preprocessed_reviews.append(sentance.strip())
In [24]:
preprocessed_reviews[1500]
Out[24]:
[3.2] Preprocessing Review Summary
In [25]:
## Similartly you can do preprocessing for review summary also
preprocessed_summary = []
# tqdm is for printing the status bar
for sentance in tqdm(final['Summary'].values):
sentance = re.sub(r"http\S+", "", sentance)
sentance = BeautifulSoup(sentance, 'lxml').get_text()
sentance = decontracted(sentance)
sentance = re.sub("\S*\d\S*", "", sentance).strip()
sentance = re.sub('[^A-Za-z]+', ' ', sentance)
# https://gist.github.com/sebleier/554280
sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
preprocessed_summary.append(sentance.strip())
In [26]:
#Splitting the data into train,CV and test
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(preprocessed_reviews,final['Score'],random_state=100,test_size=0.30,shuffle=False)
X_train2,X_CV,y_train2,y_CV = train_test_split(X_train,y_train,random_state=100,test_size=0.30,shuffle=False)
[4] Featurization¶
[4.1] BAG OF WORDS¶
In [27]:
#BoW
count_vect = CountVectorizer() #in scikit-learn
count_vect.fit(X_train2)
print("some feature names ", count_vect.get_feature_names()[:10])
print('='*50)
features_bow = count_vect.get_feature_names()
final_counts = count_vect.transform(X_train2)
print("the type of count vectorizer ",type(final_counts))
print("the shape of out text BOW vectorizer ",final_counts.get_shape())
print("the number of unique words ", final_counts.get_shape()[1])
final_countsCV = count_vect.transform(X_CV)
final_countsTEST = count_vect.transform(X_test)
[4.2] Bi-Grams and n-Grams.¶
In [28]:
#bi-gram, tri-gram and n-gram
#removing stop words like "not" should be avoided before building n-grams
# count_vect = CountVectorizer(ngram_range=(1,2))
# please do read the CountVectorizer documentation http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
# you can choose these numebrs min_df=10, max_features=5000, of your choice
count_vect2 = CountVectorizer(ngram_range=(1,2), min_df=10, max_features=5000)
count_vect2.fit(X_train2)
print("some feature names ", count_vect2.get_feature_names()[:10])
print('='*50)
features_bigrams = count_vect2.get_feature_names()
final_bigram_counts = count_vect2.transform(X_train2)
print("the type of count vectorizer ",type(final_bigram_counts))
print("the shape of out text BOW vectorizer ",final_bigram_counts.get_shape())
print("the number of unique words including both unigrams and bigrams ", final_bigram_counts.get_shape()[1])
final_bigram_countsCV = count_vect2.transform(X_CV)
final_bigram_countsTEST = count_vect2.transform(X_test)
[4.3] TF-IDF¶
In [29]:
tf_idf_vect = TfidfVectorizer(ngram_range=(1,2), min_df=10)
tf_idf_vect.fit(X_train2)
print("some sample features(unique words in the corpus)",tf_idf_vect.get_feature_names()[0:10])
print('='*50)
features_tf_idf = tf_idf_vect.get_feature_names()
final_tf_idf = tf_idf_vect.transform(X_train2)
print("the type of count vectorizer ",type(final_tf_idf))
print("the shape of out text TFIDF vectorizer ",final_tf_idf.get_shape())
print("the number of unique words including both unigrams and bigrams ", final_tf_idf.get_shape()[1])
final_tf_idfCV = tf_idf_vect.transform(X_CV)
final_tf_idfTEST = tf_idf_vect.transform(X_test)
[4.4] Word2Vec¶
In [30]:
# Train your own Word2Vec model using your own text corpus
i=0
list_of_sentence_train=[]
for sentence in X_train2:
list_of_sentence_train.append(sentence.split())
In [31]:
from gensim.models import Word2Vec
from gensim.models import KeyedVectors
w2v_model=Word2Vec(list_of_sentence_train,min_count=5,size=50, workers=4)
w2v_words = list(w2v_model.wv.vocab)
print("number of words that occured minimum 5 times ",len(w2v_words))
print("sample words ", w2v_words[0:50])
[4.4.1] Converting text into vectors using Avg W2V, TFIDF-W2V¶
[4.4.1.1] Avg W2v¶
Vectrozing Train,CV and test data text¶
In [32]:
# average Word2Vec
# compute average word2vec for each review.
sent_vectors_train = []; # the avg-w2v for each sentence/review is stored in this list
for sent in tqdm(list_of_sentence_train): # for each review/sentence
sent_vec = np.zeros(50) # as word vectors are of zero length 50, you might need to change this to 300 if you use google's w2v
cnt_words =0; # num of words with a valid vector in the sentence/review
for word in sent: # for each word in a review/sentence
if word in w2v_words:
vec = w2v_model.wv[word]
sent_vec += vec
cnt_words += 1
if cnt_words != 0:
sent_vec /= cnt_words
sent_vectors_train.append(sent_vec)
sent_vectors_train = np.array(sent_vectors_train)
print(sent_vectors_train.shape)
print(sent_vectors_train[0])
# ### Converting CV data text
i=0
list_of_sentence_cv=[]
for sentence in X_CV:
list_of_sentence_cv.append(sentence.split())
# average Word2Vec
# compute average word2vec for each review.
sent_vectors_cv = []; # the avg-w2v for each sentence/review is stored in this list
for sent in tqdm(list_of_sentence_cv): # for each review/sentence
sent_vec = np.zeros(50) # as word vectors are of zero length 50, you might need to change this to 300 if you use google's w2v
cnt_words =0; # num of words with a valid vector in the sentence/review
for word in sent: # for each word in a review/sentence
if word in w2v_words:
vec = w2v_model.wv[word]
sent_vec += vec
cnt_words += 1
if cnt_words != 0:
sent_vec /= cnt_words
sent_vectors_cv.append(sent_vec)
sent_vectors_cv = np.array(sent_vectors_cv)
print(sent_vectors_cv.shape)
print(sent_vectors_cv[0])
# Converting Test data text
i=0
list_of_sentence_test=[]
for sentence in X_test:
list_of_sentence_test.append(sentence.split())
# average Word2Vec
# compute average word2vec for each review.
sent_vectors_test = []; # the avg-w2v for each sentence/review is stored in this list
for sent in tqdm(list_of_sentence_test): # for each review/sentence
sent_vec = np.zeros(50) # as word vectors are of zero length 50, you might need to change this to 300 if you use google's w2v
cnt_words =0; # num of words with a valid vector in the sentence/review
for word in sent: # for each word in a review/sentence
if word in w2v_words:
vec = w2v_model.wv[word]
sent_vec += vec
cnt_words += 1
if cnt_words != 0:
sent_vec /= cnt_words
sent_vectors_test.append(sent_vec)
sent_vectors_test = np.array(sent_vectors_test)
print(sent_vectors_test.shape)
print(sent_vectors_test[0])
[4.4.1.2] TFIDF weighted W2v¶
In [33]:
# S = ["abc def pqr", "def def def abc", "pqr pqr def"]
model_tf = TfidfVectorizer()
tf_idf_matrix = model_tf.fit_transform(X_train2)
# we are converting a dictionary with word as a key, and the idf as a value
dictionary = dict(zip(model_tf.get_feature_names(), list(model_tf.idf_)))
For train data¶
In [34]:
# TF-IDF weighted Word2Vec
tfidf_feat = model_tf.get_feature_names() # tfidf words/col-names
# final_tf_idf is the sparse matrix with row= sentence, col=word and cell_val = tfidf
tfidf_sent_vectors_train = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in tqdm(list_of_sentence_train): # for each review/sentence
sent_vec = np.zeros(50) # as word vectors are of zero length
weight_sum =0; # num of words with a valid vector in the sentence/review
for word in sent: # for each word in a review/sentence
if word in w2v_words and word in tfidf_feat:
vec = w2v_model.wv[word]
# tf_idf = tf_idf_matrix[row, tfidf_feat.index(word)]
# to reduce the computation we are
# dictionary[word] = idf value of word in whole courpus
# sent.count(word) = tf valeus of word in this review
tf_idf = dictionary[word]*(sent.count(word)/len(sent))
sent_vec += (vec * tf_idf)
weight_sum += tf_idf
if weight_sum != 0:
sent_vec /= weight_sum
tfidf_sent_vectors_train.append(sent_vec)
row += 1
For CV data¶
In [35]:
tfidf_sent_vectors_CV = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in tqdm(list_of_sentence_cv): # for each review/sentence
sent_vec = np.zeros(50) # as word vectors are of zero length
weight_sum =0; # num of words with a valid vector in the sentence/review
for word in sent: # for each word in a review/sentence
if word in w2v_words and word in tfidf_feat:
vec = w2v_model.wv[word]
# tf_idf = tf_idf_matrix[row, tfidf_feat.index(word)]
# to reduce the computation we are
# dictionary[word] = idf value of word in whole courpus
# sent.count(word) = tf valeus of word in this review
tf_idf = dictionary[word]*(sent.count(word)/len(sent))
sent_vec += (vec * tf_idf)
weight_sum += tf_idf
if weight_sum != 0:
sent_vec /= weight_sum
tfidf_sent_vectors_CV.append(sent_vec)
row += 1
Calculating TFIDF weighted W2V for test data¶
In [36]:
tfidf_sent_vectors_test = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in tqdm(list_of_sentence_test): # for each review/sentence
sent_vec = np.zeros(50) # as word vectors are of zero length
weight_sum =0; # num of words with a valid vector in the sentence/review
for word in sent: # for each word in a review/sentence
if word in w2v_words and word in tfidf_feat:
vec = w2v_model.wv[word]
# tf_idf = tf_idf_matrix[row, tfidf_feat.index(word)]
# to reduce the computation we are
# dictionary[word] = idf value of word in whole courpus
# sent.count(word) = tf valeus of word in this review
tf_idf = dictionary[word]*(sent.count(word)/len(sent))
sent_vec += (vec * tf_idf)
weight_sum += tf_idf
if weight_sum != 0:
sent_vec /= weight_sum
tfidf_sent_vectors_test.append(sent_vec)
row += 1
In [37]:
tfidf_sent_vectors_train = np.array(tfidf_sent_vectors_train)
tfidf_sent_vectors_CV = np.array(tfidf_sent_vectors_CV)
tfidf_sent_vectors_test = np.array(tfidf_sent_vectors_test)
[5] Assignment 8: Decision Trees¶
- Apply Decision Trees on these feature sets
- SET 1:Review text, preprocessed one converted into vectors using (BOW)
- SET 2:Review text, preprocessed one converted into vectors using (TFIDF)
- SET 3:Review text, preprocessed one converted into vectors using (AVG W2v)
- SET 4:Review text, preprocessed one converted into vectors using (TFIDF W2v)
- The hyper paramter tuning (best `depth` in range [1, 5, 10, 50, 100, 500, 100], and the best `min_samples_split` in range [5, 10, 100, 500])
- Find the best hyper parameter which will give the maximum AUC value
- Find the best hyper paramter using k-fold cross validation or simple cross validation data
- Use gridsearch cv or randomsearch cv or you can also write your own for loops to do this task of hyperparameter tuning
- Graphviz
- Visualize your decision tree with Graphviz. It helps you to understand how a decision is being made, given a new vector.
- Since feature names are not obtained from word2vec related models, visualize only BOW & TFIDF decision trees using Graphviz
- Make sure to print the words in each node of the decision tree instead of printing its index.
- Just for visualization purpose, limit max_depth to 2 or 3 and either embed the generated images of graphviz in your notebook, or directly upload them as .png files.
- Feature importance
- Find the top 20 important features from both feature sets Set 1 and Set 2 using `feature_importances_` method of Decision Tree Classifier and print their corresponding feature names
- Feature engineering
- To increase the performance of your model, you can also experiment with with feature engineering like :
- Taking length of reviews as another feature.
- Considering some features from review summary as well.
- Representation of results
- You need to plot the performance of model both on train data and cross validation data for each hyper parameter, like shown in the figure.
- Once after you found the best hyper parameter, you need to train your model with it, and find the AUC on test data and plot the ROC curve on both train and test.
- Along with plotting ROC curve, you need to print the confusion matrix with predicted and original labels of test data points. Please visualize your confusion matrices using seaborn heatmaps.
- You need to plot the performance of model both on train data and cross validation data for each hyper parameter, like shown in the figure.
- Conclusion
- You need to summarize the results at the end of the notebook, summarize it in the table format. To print out a table please refer to this prettytable library link
- You need to summarize the results at the end of the notebook, summarize it in the table format. To print out a table please refer to this prettytable library link
Note: Data Leakage
- There will be an issue of data-leakage if you vectorize the entire data and then split it into train/cv/test.
- To avoid the issue of data-leakag, make sure to split your data first and then vectorize it.
- While vectorizing your data, apply the method fit_transform() on you train data, and apply the method transform() on cv/test data.
- For more details please go through this link.
Applying Decision Trees¶
[5.1] Applying Decision Trees on BOW, SET 1¶
In [38]:
from sklearn.tree import DecisionTreeClassifier
max_depth_ = [1, 5, 10, 50, 100, 500, 1000]
min_samples_split_ = [5, 10, 100, 500]
AUC_bow_train_ = []
AUC_bow_CV_ = []
for m in max_depth_:
for s in min_samples_split_:
model = DecisionTreeClassifier(max_depth=m,min_samples_split=s)
model.fit(final_counts,y_train2)
y_pred2 = model.predict_proba(final_counts)[:,1]
AUC_bow_train_.append([metrics.roc_auc_score(y_train2,y_pred2),m,s])
y_pred_CV = model.predict_proba(final_countsCV)[:,1]
AUC_bow_CV_.append([metrics.roc_auc_score(y_CV,y_pred_CV),m,s])
In [39]:
#first putting the results of each combination correspomding to max_depth in list
#then finding the max AUC for each max_depth_
'''AUC_bow_train2=[AUC_bow_train_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)] +
AUC_bow_train = [max(i)[0] for i in AUC_bow_train2]
AUC_bow_CV2 = [AUC_bow_CV_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_bow_CV = [max(i)[0] for i in AUC_bow_CV2]''''
In [40]:
#getting the parameters for which AUC score is highest
best_AUC_bow_train= max([i[0] for i in AUC_bow_train_])
for i in AUC_bow_train_:
if i[0] == best_AUC_bow_train:
optimal_depth_bow = i[1]
optimal_split_bow=i[2]
print('optimal_depth_bow ',optimal_depth_bow)
print('optimal_split_bow ',optimal_split_bow)
In [83]:
#since we are tuning two parameters and calculating AUC ,it would be better if we represent it in 3 D
import plotly.offline as offline
import plotly.graph_objs as go
offline.init_notebook_mode()
AUC_bow_train = [x[0] for x in (AUC_bow_train_)]
depth = [x[1] for x in (AUC_bow_train_)]
samples_split = [x[2] for x in (AUC_bow_train_)]
AUC_bow_CV = [x[0] for x in (AUC_bow_CV_)]
trace1 = go.Scatter3d(x=depth,y=samples_split,z=AUC_bow_train, name = 'train')
trace2 = go.Scatter3d(x=depth,y=samples_split,z=AUC_bow_CV, name = 'Cross validation')
data = [trace1, trace2]
layout = go.Layout(scene = dict(
xaxis = dict(title='n_estimators'),
yaxis = dict(title='max_depth'),
zaxis = dict(title='AUC'),))
fig = go.Figure(data=data, layout=layout)
offline.iplot(fig, filename='3d-scatter-colorscale')
[5.1.1] Top 20 important features from SET 1¶
In [42]:
# Please write all the code with proper documentation
model_DT = DecisionTreeClassifier(max_depth=optimal_depth_bow,min_samples_split=optimal_split_bow)
model_DT = model_DT.fit(final_bigram_counts,y_train2)
print(np.take(np.asarray(features_bigrams),model_DT.feature_importances_.argsort()[::-1][:20]))
[5.1.2] Graphviz visualization of Decision Tree on BOW, SET 1¶
In [43]:
# Please write all the code with proper documentation
import graphviz
from sklearn import tree
In [44]:
model_DT = DecisionTreeClassifier(max_depth=3)
model_fit = model_DT.fit(final_bigram_counts,y_train2)
In [45]:
dot_data = tree.export_graphviz(model_fit,out_file='AmazonFood.dot',feature_names=features_bigrams)
with open('AmazonFood.dot') as f:
dot_graph = f.read()
graphviz.Source(dot_graph)
Out[45]:
Testing the optimal hyperparameters¶
In [46]:
#plotting ROC curve for train and test data
model_bi = DecisionTreeClassifier(max_depth=optimal_depth_bow,min_samples_split=optimal_split_bow)
model_bi.fit(final_counts,y_train2)
y_pred_bi = model_bi.predict_proba(final_counts)[:,1]
fpr_bi,tpr_bi,t1 = metrics.roc_curve(y_train2,y_pred_bi)
y_pred_bi2 = model_bi.predict_proba(final_countsTEST)[:,1]
fpr_bi2,tpr_bi2,t2 = metrics.roc_curve(y_test,y_pred_bi2)
plt.plot(fpr_bi,tpr_bi,label='Train ROC Curve=' +str(metrics.auc(fpr_bi,tpr_bi)))
plt.plot(fpr_bi2,tpr_bi2,label='Test ROC Curve=' +str(metrics.auc(fpr_bi2,tpr_bi2)))
plt.legend()
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve of train and test data')
plt.show()
In [47]:
#function to calculate the threshold value with high tpr and low fpr
def find_best_threshold(thresh, fpr, tpr):
t = thresh[np.argmax(tpr*(1-fpr))]
# (tpr*(1-fpr)) will be maximum if your fpr is very low and tpr is very high
print("the maximum value of tpr*(1-fpr)", max(tpr*(1-fpr)), "for threshold", np.round(t,3))
return t
#function to get the predictions with best threshold
def predict_with_best_t(proba, thresh):
predictions = []
for i in proba:
if i>=thresh:
predictions.append(1)
else:
predictions.append(0)
return predictions
In [48]:
#Finding the threshold value of probability on train data and then showng results on test data
best_t = find_best_threshold(t1,fpr_bi,tpr_bi)
print("*********Train confusion matrix*********")
print(confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t)))
cm_tr = confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_tr,annot=True,ax=ax2,fmt='g')
Out[48]:
In [49]:
print("*********Test confusion matrix*********")
print(confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t)))
cm_test = confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_test,annot=True,ax=ax2,fmt='g')
Out[49]:
[5.2] Applying Decision Trees on TFIDF, SET 2¶
In [50]:
from sklearn.tree import DecisionTreeClassifier
max_depth_ = [1, 5, 10, 50, 100, 500, 1000]
min_samples_split_ = [5, 10, 100, 500]
AUC_tfidf_train_ = []
AUC_tfidf_CV_ = []
for m in max_depth_:
for s in min_samples_split_:
model = DecisionTreeClassifier(max_depth=m,min_samples_split=s)
model.fit(final_tf_idf,y_train2)
y_pred2 = model.predict_proba(final_tf_idf)[:,1]
AUC_tfidf_train_.append([metrics.roc_auc_score(y_train2,y_pred2),m,s])
y_pred_CV = model.predict_proba(final_tf_idfCV)[:,1]
AUC_tfidf_CV_.append([metrics.roc_auc_score(y_CV,y_pred_CV),m,s])
In [51]:
#first putting the results of each combination correspomding to max_depth in list
#then finding the max AUC for each max_depth_
'''AUC_tfidf_train2=[AUC_tfidf_train_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_tfidf_train = [max(i)[0] for i in AUC_tfidf_train2]
AUC_tfidf_CV2 = [AUC_tfidf_CV_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_tfidf_CV = [max(i)[0] for i in AUC_tfidf_CV2]'''
In [85]:
#since we are tuning two parameters and calculating AUC ,it would be better if we represent it in 3 D
import plotly.offline as offline
import plotly.graph_objs as go
offline.init_notebook_mode()
AUC_tfidf_train = [x[0] for x in (AUC_tfidf_train_)]
depth_tfidf = [x[1] for x in (AUC_tfidf_train_)]
samples_split_tfidf = [x[2] for x in (AUC_tfidf_train_)]
AUC_tfidf_CV = [x[0] for x in (AUC_tfidf_CV_)]
trace1 = go.Scatter3d(x=depth_tfidf,y=samples_split_tfidf,z=AUC_tfidf_train, name = 'train')
trace2 = go.Scatter3d(x=depth_tfidf,y=samples_split_tfidf,z=AUC_tfidf_CV, name = 'Cross validation')
data = [trace1, trace2]
layout = go.Layout(scene = dict(
xaxis = dict(title='n_estimators'),
yaxis = dict(title='max_depth'),
zaxis = dict(title='AUC'),))
fig = go.Figure(data=data, layout=layout)
offline.iplot(fig, filename='3d-scatter-colorscale')
In [53]:
#getting the parameters for which AUC score is highest
best_AUC_tfidf_train= max([i[0] for i in AUC_tfidf_train_])
for i in AUC_tfidf_train_:
if i[0] == best_AUC_tfidf_train:
optimal_depth_tfidf = i[1]
optimal_split_tfidf=i[2]
print('optimal_depth_tfidf ',optimal_depth_tfidf)
print('optimal_split_tfidf ',optimal_split_tfidf)
[5.2.1] Top 20 important features from SET 2¶
In [54]:
# Please write all the code with proper documentation
model_DT = DecisionTreeClassifier(max_depth=optimal_depth_tfidf,min_samples_split=optimal_split_tfidf)
model_DT = model_DT.fit(final_bigram_counts,y_train2)
print(np.take(np.asarray(features_bigrams),model_DT.feature_importances_.argsort()[::-1][:20]))
[5.2.2] Graphviz visualization of Decision Tree on TFIDF, SET 2¶
In [55]:
# Please write all the code with proper documentation
# Please write all the code with proper documentation
import graphviz
from sklearn import tree
model_DT = DecisionTreeClassifier(max_depth=3)
model_fit = model_DT.fit(final_tf_idf,y_train2)
dot_data = tree.export_graphviz(model_fit,out_file='AmazonFood.dot',feature_names=features_tf_idf)
with open('AmazonFood.dot') as f:
dot_graph = f.read()
graphviz.Source(dot_graph)
Out[55]:
In [56]:
#plotting ROC curve for train and test data
model_bi = DecisionTreeClassifier(max_depth=optimal_depth_tfidf,min_samples_split=optimal_split_tfidf)
model_bi.fit(final_tf_idf,y_train2)
y_pred_bi = model_bi.predict_proba(final_tf_idf)[:,1]
fpr_train,tpr_train,t1 = metrics.roc_curve(y_train2,y_pred_bi)
y_pred_bi2 = model_bi.predict_proba(final_tf_idfTEST)[:,1]
fpr_test,tpr_test,t2 = metrics.roc_curve(y_test,y_pred_bi2)
plt.plot(fpr_train,tpr_train,label='Train ROC Curve=' +str(metrics.auc(fpr_train,tpr_train)))
plt.plot(fpr_test,tpr_test,label='Test ROC Curve=' +str(metrics.auc(fpr_test,tpr_test)))
plt.legend()
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve of train and test data')
plt.show()
In [57]:
#Finding the threshold value of probability on train data and then showng results on test data
best_t = find_best_threshold(t1,fpr_train,tpr_train)
print("*********Train confusion matrix*********")
print(confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t)))
cm_tr = confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_tr,annot=True,ax=ax2,fmt='g')
Out[57]:
In [58]:
print("*********Test confusion matrix*********")
print(confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t)))
cm_test = confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_test,annot=True,ax=ax2,fmt='g')
Out[58]:
[5.3] Applying Decision Trees on AVG W2V, SET 3¶
In [59]:
# Please write all the code with proper documentation
from sklearn.tree import DecisionTreeClassifier
max_depth_ = [1, 5, 10, 50, 100, 500, 1000]
min_samples_split_ = [5, 10, 100, 500]
AUC_avgw2v_train_ = []
AUC_avgw2v_CV_ = []
for m in max_depth_:
for s in min_samples_split_:
model = DecisionTreeClassifier(max_depth=m,min_samples_split=s)
model.fit(sent_vectors_train,y_train2)
y_pred2 = model.predict_proba(sent_vectors_train)[:,1]
AUC_avgw2v_train_.append([metrics.roc_auc_score(y_train2,y_pred2),m,s])
y_pred_CV = model.predict_proba(sent_vectors_cv)[:,1]
AUC_avgw2v_CV_.append([metrics.roc_auc_score(y_CV,y_pred_CV),m,s])
In [60]:
#first putting the results of each combination correspomding to max_depth in list
#then finding the max AUC for each max_depth_
'''AUC_avgw2v_train2=[AUC_avgw2v_train_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_avgw2v_train = [max(i)[0] for i in AUC_avgw2v_train2]
AUC_avgw2v_CV2 = [AUC_avgw2v_CV_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_avgw2v_CV = [max(i)[0] for i in AUC_avgw2v_CV2]'''
In [87]:
#since we are tuning two parameters and calculating AUC ,it would be better if we represent it in 3 D
import plotly.offline as offline
import plotly.graph_objs as go
offline.init_notebook_mode()
AUC_avgw2v_train = [x[0] for x in (AUC_avgw2v_train_)]
depth_avgw2v = [x[1] for x in (AUC_avgw2v_train_)]
samples_split_avgw2v = [x[2] for x in (AUC_avgw2v_train_)]
AUC_avgw2v_CV = [x[0] for x in (AUC_avgw2v_CV_)]
trace1 = go.Scatter3d(x=depth_avgw2v,y=samples_split_avgw2v,z=AUC_avgw2v_train, name = 'train')
trace2 = go.Scatter3d(x=depth_avgw2v,y=samples_split_avgw2v,z=AUC_avgw2v_CV, name = 'Cross validation')
data = [trace1, trace2]
layout = go.Layout(scene = dict(
xaxis = dict(title='n_estimators'),
yaxis = dict(title='max_depth'),
zaxis = dict(title='AUC'),))
fig = go.Figure(data=data, layout=layout)
offline.iplot(fig, filename='3d-scatter-colorscale')
In [61]:
#getting the parameters for which AUC score is highest
best_AUC_avgw2v_train= max([i[0] for i in AUC_avgw2v_train_])
for i in AUC_avgw2v_train_:
if i[0] == best_AUC_avgw2v_train:
optimal_depth_avgw2v = i[1]
optimal_split_avgw2v=i[2]
print('optimal_depth_avgw2v ',optimal_depth_avgw2v)
print('optimal_split_avgw2v ',optimal_split_avgw2v)
Testing the model
In [63]:
final_model_avgw2v = DecisionTreeClassifier(max_depth=optimal_depth_avgw2v,min_samples_split=optimal_split_avgw2v)
final_model_avgw2v.fit(sent_vectors_train,y_train2)
y_pred_bi = final_model_avgw2v.predict_proba(sent_vectors_train)[:,1]
fpr_train,tpr_train,t1 = metrics.roc_curve(y_train2,y_pred_bi)
y_pred_bi2 = final_model_avgw2v.predict_proba(sent_vectors_test)[:,1]
fpr_test,tpr_test,t2 = metrics.roc_curve(y_test,y_pred_bi2)
plt.title('Plot of True positive VS False Positive for AVG W2V data')
plt.plot(fpr_train,tpr_train,label='Train ROC Curve=' +str(metrics.auc(fpr_train,tpr_train)))
plt.plot(fpr_test,tpr_test,label='Test ROC Curve=' +str(metrics.auc(fpr_test,tpr_test)))
plt.legend()
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve of train and test data')
plt.show()
In [64]:
#Finding the threshold value of probability on train data and then showng results on test data
best_t = find_best_threshold(t1,fpr_train,tpr_train)
print("*********Train confusion matrix*********")
print(confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t)))
cm_tr = confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_tr,annot=True,ax=ax2,fmt='g')
Out[64]:
In [65]:
print("*********Test confusion matrix*********")
print(confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t)))
cm_test = confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_test,annot=True,ax=ax2,fmt='g')
Out[65]:
[5.4] Applying Decision Trees on TFIDF W2V, SET 4¶
In [66]:
# Please write all the code with proper documentation
from sklearn.tree import DecisionTreeClassifier
max_depth_ = [1, 5, 10, 50, 100, 500, 1000]
min_samples_split_ = [5, 10, 100, 500]
AUC_tfw2v_train_ = []
AUC_tfw2v_CV_ = []
for m in max_depth_:
for s in min_samples_split_:
model = DecisionTreeClassifier(max_depth=m,min_samples_split=s)
model.fit(tfidf_sent_vectors_train,y_train2)
y_pred2 = model.predict_proba(tfidf_sent_vectors_train)[:,1]
AUC_tfw2v_train_.append([metrics.roc_auc_score(y_train2,y_pred2),m,s])
y_pred_CV = model.predict_proba(sent_vectors_cv)[:,1]
AUC_tfw2v_CV_.append([metrics.roc_auc_score(y_CV,y_pred_CV),m,s])
In [67]:
#first putting the results of each combination correspomding to max_depth in list
#then finding the max AUC for each max_depth_
'''AUC_tfw2v_train2=[AUC_tfw2v_train_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_tfw2v_train = [max(i)[0] for i in AUC_tfw2v_train2]
AUC_tfw2v_CV2 = [AUC_tfw2v_CV_[i:i+4] for i in range(0,len(max_depth_) * len(min_samples_split_),4)]
AUC_tfw2v_CV = [max(i)[0] for i in AUC_tfw2v_CV2]'''
In [89]:
#since we are tuning two parameters and calculating AUC ,it would be better if we represent it in 3 D
import plotly.offline as offline
import plotly.graph_objs as go
offline.init_notebook_mode()
AUC_tfw2v_train = [x[0] for x in (AUC_tfw2v_train_)]
depth_tfw2v = [x[1] for x in (AUC_tfw2v_train_)]
samples_split_tfw2v = [x[2] for x in (AUC_tfw2v_train_)]
AUC_tfw2v_CV = [x[0] for x in (AUC_tfw2v_CV_)]
trace1 = go.Scatter3d(x=depth_tfw2v,y=samples_split_tfw2v,z=AUC_tfw2v_train, name = 'train')
trace2 = go.Scatter3d(x=depth_tfw2v,y=samples_split_tfw2v,z=AUC_tfw2v_CV, name = 'Cross validation')
data = [trace1, trace2]
layout = go.Layout(scene = dict(
xaxis = dict(title='n_estimators'),
yaxis = dict(title='max_depth'),
zaxis = dict(title='AUC'),))
fig = go.Figure(data=data, layout=layout)
offline.iplot(fig, filename='3d-scatter-colorscale')
In [69]:
#getting the parameters for which AUC score is highest
best_AUC_tfw2v_train= max([i[0] for i in AUC_tfw2v_train_])
for i in AUC_tfw2v_train_:
if i[0] == best_AUC_tfw2v_train:
optimal_depth_tfw2v = i[1]
optimal_split_tfw2v=i[2]
print('optimal_depth_tfw2v ',optimal_depth_tfw2v)
print('optimal_split_tfw2v ',optimal_split_tfw2v)
In [70]:
final_model_tfw2v = DecisionTreeClassifier(max_depth=optimal_depth_tfw2v,min_samples_split=optimal_split_tfw2v)
final_model_tfw2v.fit(tfidf_sent_vectors_train,y_train2)
y_pred_bi = final_model_tfw2v.predict_proba(tfidf_sent_vectors_train)[:,1]
fpr_train,tpr_train,t1 = metrics.roc_curve(y_train2,y_pred_bi)
y_pred_bi2 = final_model_tfw2v.predict_proba(tfidf_sent_vectors_test)[:,1]
fpr_test,tpr_test,t2 = metrics.roc_curve(y_test,y_pred_bi2)
plt.title('Plot of True positive VS False Positive for AVG W2V data')
plt.plot(fpr_train,tpr_train,label='Train ROC Curve=' +str(metrics.auc(fpr_train,tpr_train)))
plt.plot(fpr_test,tpr_test,label='Test ROC Curve=' +str(metrics.auc(fpr_test,tpr_test)))
plt.legend()
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curve of train and test data')
plt.show()
In [71]:
#Finding the threshold value of probability on train data and then showng results on test data
best_t = find_best_threshold(t1,fpr_train,tpr_train)
print("*********Train confusion matrix*********")
print(confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t)))
cm_tr = confusion_matrix(y_train2, predict_with_best_t(y_pred_bi, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_tr,annot=True,ax=ax2,fmt='g')
Out[71]:
In [72]:
print("*********Test confusion matrix*********")
print(confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t)))
cm_test = confusion_matrix(y_test, predict_with_best_t(y_pred_bi2, best_t))
plt.figure()
ax2 = plt.subplot()
sns.heatmap(cm_test,annot=True,ax=ax2,fmt='g')
Out[72]:
[6] Conclusions¶
In [73]:
# Please compare all your models using Prettytable library
print('Number of data points used for Decision tree model : 50k\n')
from prettytable import PrettyTable
x = PrettyTable()
x.field_names = ["Vectorizer","Max Depth","Min Samples Split" ,"AUC"]
x.add_row(["BOW",optimal_depth_bow,optimal_split_bow,best_AUC_bow_train])
x.add_row(["TFIDF",optimal_depth_bow,optimal_split_bow,best_AUC_tfidf_train])
x.add_row(["Avg W2v",optimal_depth_avgw2v,optimal_split_avgw2v,best_AUC_avgw2v_train])
x.add_row(["TFIDF Avg W2v",optimal_depth_tfw2v,optimal_split_tfw2v,best_AUC_tfw2v_train])
print(x)
Conclusion¶
- Total number of data points used for this assignment is 50K
- The models seem to be overfit as they are returning very good AUC score for train data but poor values for test data.One way to avoid overfitting could be use more data for training.Another one could be to fed less number of parameters.Random forest model is able to overcome these issues and performing really well for this scenario.